Surface reconstruction (SR) involves identifying surfaces of objects in a physical space. For example, the surfaces can represents walls, floors, a ceiling, furniture or other objects in a room. The surfaces provide a model of the physical environment. Such a model can be used in a wide range of applications. One example is guiding a robot through a physical space. Another example is displaying a virtual object to a user, such by using a head mounted display device.